Automated remote car counting

ABSTRACT

A system for automated car counting comprises a satellite-based image collection subsystem; a data storage subsystem; and an analysis software module stored and operating on a computer coupled to the data storage subsystem. The satellite-based image collection subsystem collects images corresponding to a plurality of areas of interest and stores them in the data storage subsystem. The analysis module: (a) retrieves images corresponding to an area of interest from the data storage subsystem; (b) identifies a parking space in an image; (c) determines if there is a car located in the parking space; (d) determines a location, size, and angular direction of a car in a parking space; (e) determines an amount of overlap of a car with an adjacent parking space; (f) iterates steps (b)-(e) until no unprocessed parking spaces remain; and (g) iterates steps (a)-(f) until no unprocessed images corresponding to areas of interest remain.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/942,544, titled “AUTOMATED REMOTE CAR COUNTING”, filed on Jul. 15,2013, the entire specification of which is incorporated hereby byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to the field of digital image processing, and moreparticularly to the field of analysis of remote sensing satellite and/oraerial image data.

2. Discussion of the State of Prior Art

In many cases, the number of generic cars (by which is meant any motorvehicle, car, SUV, pickup truck, etc., with 2 axles and 3 or morewheels) and the available designated parking slots, or spaces bothobservable by a remote sensing instrument on a satellite and/or aerialimaging system on an aircraft may need to be known or estimated bypersons or entities other than those responsible for the facilities inquestion. For instance, it might be of value to know, on atime-sensitive basis, whether cars have been added or withdrawn fromspecific parking facilities. Obtaining such information typically is notpossible on an ad hoc basis, either because the information isproprietary (private parking facilities, or public facilities withassociated storage), or because it is released publicly on a fixedschedule (which is the case for public or municipal parking lots).Satellite imaging and other remote sensing capabilities make it possibleto remotely observe such facilities independently of the facilitiesthemselves (or of their owners and operators), as long as they areviewable from the remote sensing platform. Obviously, in the case ofmultistoried parking facilities, only the upper-most level would beobservable from overhead.

Given a source of remotely observed parking facilities such as shortterm and/or long term parking and/or storage lots, what is needed is ameans for automatically estimating the number of cars parked and/orstored therein without the involvement of the operators or owners ofthose facilities. Additionally, the ratio of cars to parking slots (anoccupancy ratio) may be of interest. In some cases, the variation in thenumber of cars parked and/or stored over specific time periods may bethe end product.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, in apreferred embodiment of the invention, various systems and methods foranalysis of remotely sensed image data of parking facilities and/or carstorage lots, for estimating the number of spaces or slots containedtherein, and last, for estimating the number of cars parked and/orstored therein and monitoring them over time.

This Patent Application describes a system for automated orsemi-automated estimation of the number of cars parked/stored in anassumed set of polygonal parking lots. Further, it can be assumed thatthe parking/storage lot is marked off with individual rows of parkingslots or spaces. This data may be provided manually via trainingalgorithms, through existing data sets such as OPENSTREETMAP™ or throughan automated detection algorithms such as an automated parking lot/rowdetector.

The preferred embodiment of this invention creates a novelspecial-purpose analysis system from a mix of newly written algorithmsin addition to making use of prior art analysis software with speciallyselected parameters and combined in non-obvious and novel ways. Thisanalysis system is designed to automatically or semi-automaticallyanalyze remotely sensed digital images, to locate parking spaces orslots, and to detect and count cars in those spaces or slots inparking/storage facilities (those that are visible from overhead).

According to an embodiment, the system may be implemented on, but notlimited to, stand-alone high-end image-analysis workstations, or acloud-based workstation with a web browser to interact with the variousfunctions. Any implementation would always support a manual method whereelements in an image are tagged for analysis by an automaticbatch-oriented method, or an interactive semi-automatic method thatrefines an automated technique via machine learning.

According to a preferred embodiment of the invention, a system forautomated car counting is disclosed, comprising a satellite-based imagecollection subsystem; a data storage subsystem; and an analysis softwaremodule stored and operating on a computer coupled to the data storagesubsystem. The satellite-based image collection subsystem collectsimages corresponding to a plurality of areas of interest and stores themin the data storage subsystem. The analysis module: (a) retrieves imagescorresponding to an area of interest from the data storage subsystem;(b) identifies a parking space in an image; (c) determines if there is acar located in the parking space; (d) determines a location, size, andangular direction of a car in a parking space; (e) determines an amountof overlap of a car with an adjacent parking space; (f) iterates steps(b)-(e) until no unprocessed parking spaces remain; and (g) iteratessteps (a)-(f) until no unprocessed images corresponding to areas ofinterest remain.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of theinvention, and together with the descriptions, serve to explain theprinciples of the invention according to embodiments. One skilled in theart will recognize that the particular embodiments illustrated in thedrawings are merely exemplary, and are not intended to limit the scopeof the present invention.

FIG. 1 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device.

FIG. 2 is a block diagram illustrating an exemplary logical architecturefor a client device, according to an embodiment of the invention.

FIG. 3 is a block diagram illustrating an exemplary architecturalarrangement of clients, according to an embodiment of the invention.

FIG. 4 is a block diagram providing a conceptual overview of a preferredembodiment of the invention, by which parking lot capacity and number ofcars parked therein may be estimated.

FIG. 5 is a block diagram of a system for estimating parking lotcapacity and number of cars parked therein using a cloud-based service,according to a preferred embodiment of the invention.

FIG. 6 is a process flow diagram of a method for estimating parking lotcapacity and number of cars parked therein using semi-automatedanalysis, according to a preferred embodiment of the invention.

FIG. 7 is a process flow diagram of a further method for estimatingparking lot capacity and number of cars parked therein usingsemi-automated analysis, according to an embodiment of the invention.

FIG. 8 is a process flow diagram of a further method for Post DetectionOptimization of parking lot capacity and number of cars parked thereinusing semi-automated analysis, according to an embodiment of theinvention.

DETAILED DESCRIPTION OF EMBODIMENTS

The inventor has conceived, and reduced to practice, various systems andmethods for automated or semi-automated estimation of the number of carsin parking and/or storage lots by analysis of digital image data fromremote sensing instrumentation. Of interest is the ability to monitorvehicular traffic that may be very dynamic. The remote sensinginstrument may be part of an orbital satellite payload, or part of apayload on an aircraft flying over areas of ground with car parking orstorage lots visible from overhead.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beunderstood that these are presented for illustrative purposes only. Thedescribed embodiments are not intended to be limiting in any sense. Oneor more of the inventions may be widely applicable to numerousembodiments, as is readily apparent from the disclosure. In general,embodiments are described in sufficient detail to enable those skilledin the art to practice one or more of the inventions, and it is to beunderstood that other embodiments may be utilized and that structural,logical, software, electrical and other changes may be made withoutdeparting from the scope of the particular inventions. Accordingly,those skilled in the art will recognize that one or more of theinventions may be practiced with various modifications and alterations.

Particular features of one or more of the inventions may be describedwith reference to one or more particular embodiments or figures thatform a part of the present disclosure, and in which are shown, by way ofillustration, specific embodiments of one or more of the inventions. Itshould be understood, however, that such features are not limited tousage in the one or more particular embodiments or figures withreference to which they are described. The present disclosure is neithera literal description of all embodiments of one or more of theinventions nor a listing of features of one or more of the inventionsthat must be present in all embodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries, logical or physical.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions. Similarly, although process steps, method steps,analysis steps or the like may be described in a sequential order, suchprocesses, methods and analysis steps may generally be configured towork in alternate orders, unless specifically stated to the contrary. Inother words, any sequence or order of steps that may be described inthis patent application does not, in and of itself, indicate arequirement that the steps be performed in that order. The steps ofdescribed processes may be performed in any order practical. Further,some steps may be performed simultaneously despite being described orimplied as occurring non-simultaneously (e.g., because one step isdescribed after the other step).

Moreover, the illustration of a process by its depiction in a drawingdoes not imply that the illustrated process is exclusive of othervariations and modifications thereto, does not imply that theillustrated process or any of its steps are necessary to one or more ofthe invention(s), and does not imply that the illustrated process ispreferred. Also, steps are generally described once per embodiment, butthis does not mean they must occur once, or that they may only occuronce each time a process, method, or analysis is carried out orexecuted. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described, it will be readily apparent that a single deviceor article may be used in place of the more than one device or article.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should be notedthat particular embodiments include multiple iterations of a techniqueor multiple instantiations of a mechanism unless noted otherwise.Process descriptions or blocks in figures should be understood asrepresenting modules, segments, or portions of code which include one ormore executable instructions for implementing specific logical functionsor steps in the process. Alternate implementations are included withinthe scope of embodiments of the present invention in which, for example,functions may be executed out of order from that shown or discussed,including substantially concurrently or in reverse order, depending onthe functionality involved, as would be understood by those havingordinary skill in the art.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card. Referring to FIG. 2, inorder to allow interaction with the digital image data, a visual displaysystem (e.g. output device 260), a means of generating a digital“cursor” that can be moved through the image and it's position read out,and a hand operated tracking/pointing device (e.g. input device 270 maybe of any type suitable for receiving user input), including for examplea keyboard, touchscreen, microphone (for example, for voice input),mouse, touchpad, trackball, or any combination thereof that can controlthe cursor's position, or via some other hardware-only approach known inthe art.

Software/hardware hybrid implementations of at least some of anembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory.

Such network devices may have multiple network interfaces that may beconfigured or designed to utilize different types of networkcommunication protocols. A general architecture for some of thesemachines may be disclosed herein in order to illustrate one or moreexemplary means by which a given unit of functionality may beimplemented. According to specific embodiments, at least some of thefeatures or functionalities of the various embodiments disclosed hereinmay be implemented on one or more general-purpose computers associatedwith one or more networks, such as for example an end-user computersystem, a client computer, a network server or other server system, amobile computing device (e.g., tablet computing device, mobile phone,smartphone, laptop, and the like), a consumer electronic device, a musicplayer, or any other suitable electronic device, router, switch, or thelike, or any combination thereof.

In at least some embodiments, at least some of the features orfunctionalities of the various embodiments disclosed herein may beimplemented in one or more virtualized computing environments (e.g.,network computing clouds, virtual machines hosted on one or morephysical computing machines, or the like). Moreover, in some embodimentsone or more aspects, or all aspects, of the invention may optionally beimplemented via a specially programmed chip (for instance, anapplication specific integrated circuit, or ASIC, or an erasableprogrammable read only memory, or EPROM), or via some otherhardware-only approach known in the art.

Referring now to FIG. 1, there is shown a block diagram depicting anexemplary computing device 100 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 100 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 100 may be adaptedto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork, a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 100 includes one or more centralprocessing units (CPU) 102, one or more interfaces 110, and one or morebusses 106 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 102may be responsible for implementing specific functions associated withthe functions of a specifically configured computing device or machine.For example, in at least one embodiment, a computing device 100 may beconfigured or designed to function as a server system utilizing CPU 102,local memory 101 and/or remote memory 120, and interface(s) 110. In atleast one embodiment, CPU 102 may be caused to perform one or more ofthe different types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate application software, drivers, andthe like.

CPU 102 may include one or more processors 103 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 103 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 100.

In a specific embodiment, a local memory 101 (such as non-volatilerandom access memory (RAM) and/or read-only memory (ROM), including forexample one or more levels of cached memory) may also form part of CPU102. However, there are many different ways in which memory may becoupled to system 100. Memory 101 may be used for a variety of purposessuch as, for example, caching and/or storing data, programminginstructions, and the like.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 110 are provided as network interfacecards (NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 110 may forexample support other peripherals used with computing device 100. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, Firewire™, PCI, parallel, radio frequency (RF),Bluetooth™ near-field communications (e.g., using near-field magnetics),802.11 (WiFi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces,Gigabit Ethernet interfaces, asynchronous transfer mode (ATM)interfaces, high-speed serial interface (HSSI) interfaces, Point of Sale(POS) interfaces, fiber data distributed interfaces (FDDIs), and thelike. Generally, such interfaces 110 may include ports appropriate forcommunication with appropriate media. In some cases, they may alsoinclude an independent processor and, in some in stances, volatileand/or non-volatile memory (e.g., RAM).

Although the system shown in FIG. 1 illustrates one specificarchitecture for a computing device 100 for implementing one or moreembodiments of the inventions described herein, it is by no means theonly device architecture on which at least a portion of the features andtechniques described herein may be implemented. For example,architectures having one or any number of processors 103 may be used,and such processors 103 may be present in a single device or distributedamong any number of devices. In one embodiment, a single processor 103handles communications as well as routing computations, while in otherembodiments a separate dedicated communications processor may beprovided. In various embodiments, different types of features orfunctionalities may be implemented in a system according to theinvention that includes a client device (such as a tablet device orsmartphone running client software) and server systems (such as a serversystem described in more detail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 120 and local memory 101) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of anembodiments described herein (or any combinations of the above).

Program instructions may control execution of or comprise an operatingsystem and/or one or more applications, for example. Memory 120 ormemories 101, 120 may also be configured to store data structures,configuration data, encryption data, historical system operationsinformation, or any other specific or generic non-program informationdescribed herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include non-transitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein.

Examples of such non-transitory machine-readable storage media include,but are not limited to, magnetic media such as hard disks, floppy disks,and magnetic tape; optical media such as CD-ROM disks; magneto-opticalmedia such as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory, solid state drives, memristormemory, random access memory (RAM), and the like.

Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a Java™ compiler and may be executed using a Java virtualmachine or equivalent, or files containing higher level code that may beexecuted by the computer using an interpreter (for example, scriptswritten in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 2,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 200 includes processors210 that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 230. Processors 210 may carry out computinginstructions under control of an operating system 220 such as, forexample, a version of Microsoft's Windows™ operating system, Apple's MacOS/X or iOS operating systems, varieties of the Linux operating system,Google's Android™ operating system, or the like.

In many cases, one or more shared services 225 may be operable in system200, and may be useful for providing common services to clientapplications 230. Services 225 may for example be Windows™ services,user-space common services in a Linux environment, or any other type ofcommon service architecture used with operating system 210. Inputdevices 270 may be of any type suitable for receiving user input,including for example a keyboard, touchscreen, microphone (for example,for voice input), mouse, touchpad, trackball, or any combinationthereof. Output devices 260 may be of any type suitable for providingoutput to one or more users, whether remote or local to system 200, andmay include for example one or more screens for visual output, speakers,printers, or any combination thereof.

Memory 240 may be random-access memory having any structure andarchitecture known in the art, for use by processors 210, for example torun software. Storage devices 250 may be any magnetic, optical,mechanical, memristor, or electrical storage device for storage of datain digital form. Examples of storage devices 250 include flash memory,magnetic hard drive, CD-ROM, and/or the like.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 3, there is shown a blockdiagram depicting an exemplary architecture for implementing at least aportion of a system according to an embodiment of the invention on adistributed computing network.

According to an embodiment, any number of clients 330 may beaccommodated. Each client 330 may run software for implementingclient-side portions of the present invention; clients may comprise asystem 200 such as that illustrated in FIG. 2. In addition, any numberof servers 320 may be provided for handling requests received from oneor more clients 330.

Clients 330 and servers 320 may communicate with one another via one ormore electronic networks 310, which may be in various embodiments any ofthe Internet, a wide area network, a mobile telephony network, awireless network (such as WiFi, Wimax™, and so forth), or a local areanetwork (or indeed any network topology known in the art; the inventiondoes not prefer any one network topology over any other). Networks 310may be implemented using any known network protocols, including forexample wired and/or wireless protocols.

In addition, in some embodiments, servers 320 may call external services370 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 370 may take place, for example, via one or morenetworks 310. In various embodiments, external services 370 may compriseweb-enabled services or functionality related to or installed on thehardware device itself.

For example, in an embodiment where client applications 230 areimplemented on a smartphone or other electronic device, clientapplications 230 may obtain information stored in a server system 320 inthe cloud or on an external service 370 deployed on one or more of aparticular enterprise's or user's premises.

In some embodiments of the invention, clients 330 or servers 320 (orboth) may make use of one or more specialized services or appliancesthat may be deployed locally or remotely across one or more networks310. For example, one or more databases 340 may be used or referred toby one or more embodiments of the invention.

It should be understood by one having ordinary skill in the art thatdatabases 340 may be arranged in a wide variety of architectures andusing a wide variety of data access and manipulation means. For example,in various embodiments one or more databases 340 may comprise arelational database system using a structured query language (SQL),while others may comprise an alternative data storage technology such asthose referred to in the art as “NoSQL” (for example, Hadoop, MapReduce,BigTable, and so forth). In some embodiments, variant databasearchitectures such as column-oriented databases, in-memory databases,clustered databases, distributed databases, or even flat file datarepositories may be used according to the invention.

It will be appreciated by one having ordinary skill in the art that anycombination of known or future database technologies may be used asappropriate, unless a specific database technology or a specificarrangement of components is specified for a particular embodimentherein. Moreover, it should be appreciated that the term “database” asused herein may refer to a physical database machine, a cluster ofmachines acting as a single database system, or a logical databasewithin an overall database management system. Unless a specific meaningis specified for a given use of the term “database”, it should beconstrued to mean any of these senses of the word, all of which areunderstood as a plain meaning of the term “database” by those havingordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 360 and configuration systems 350. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 360 orconfiguration 350 system or approach is specifically required by thedescription of any specific embodiment.

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules can be variously implemented torun on server and/or client components.

Conceptual Overview

According to an embodiment of the invention, FIG. 4 is a block diagramproviding a conceptual overview of a system 400 for determining featuresor characteristics of cars in parking lots. As illustrated, variouscomponents of system 400 may be interconnected via a variety of meansfor communication, such as via a direct physical connection (such as viaelectronic cables or wired connections), or via a remote or cloud-basedcommunication connection interchangeably. In such a manner, it will beappreciated that system 400 may be readily adaptable to a variety ofconfigurations comprising varying arrangements of distributed or localcomponents and arrangements, as described later (referring to FIG. 5).

According to an embodiment, a pointing device 410 may be used toposition a cursor 411 to make an interactive selection of a car 412displayed on a display device 420, which might be a computer monitor,electronic device with integral display such as a tablet or laptopcomputing device, or any similar video display device. Using such adisplay 420 with appropriate software components for interaction (asdescribed below, referring to FIG. 5) it may be possible to determineinitial location coordinates for a car 412. Then cursor 411 may be usedto interactively determine initial values for a car, such as (forexample) its orientation.

Alternatively, car dimensions may be loaded from an existing database430 of known parking facility dimensions, which may either be acquiredfrom parking facility operators or determined directly by a user of thisinvention and stored for future use. Furthermore, while most examplesherein are made with reference to parked cars, it will be apparent toone having ordinary skill in the art that other types of vehicles may beused according to the invention. Also, most examples are made regardingcar parking facilities, but it will be readily understood by one havingordinary skill in the art that car parking/storage is merely anexemplary use of the invention, and many other uses of the invention areenvisioned by the inventor.

For example, aircraft at an airport, containerized storage on board atrain, containers on a ship; dynamic movement of any of the above vs.time of any period over which you have remotely-obtained images,according to various embodiments of the invention.

According to an embodiment, extraction module 440 may extract imageacquisition parameters and sun position from an image data header.Analysis module 450 may use those parameters together with the initialphysical parameters for a car (orientation, and 3-D location) determinedby application of an interactive cursor to determine the relative imagerotation angle for a parking lot row (or, as mentioned above, loadeddirectly from an existing database of parking lot dimensions).

In the art, parameters may be extracted from the image data header foreach digital image by extraction module 440, providing, for example, atime of observation, an orbital position of the satellite observingplatform, and an orientation of the satellite observing platform(R=roll, P=pitch, & Y=yaw) with respect to an earth-centered coordinatesystem.

In the art, the ancillary data for each digital image may also provideimage acquisition parameters in order that image distance measurements(in pixels) may be scaled to absolute linear measurements. Ancillarydata for each digital image may also provide sun angle position data forthe time corresponding to the center of the digital image, from whichanalysis module 450 calculates shadow direction, length, and shape forthe digital images of the parking facility.

Initial estimates may be available for some car's location, andorientation. These initial estimates may be manually determined fromother remote sensing sources such as aerial or satellite imagery, orextracted from the digital image through some other means, such as byuse of a trackball controlled cursor, or they may be loaded directlyfrom an existing data store.

According to the embodiment illustrated, output 460 from variouscomponents of system 400 may be produced and transmitted via aconnection (which may be of varied nature as described above), such asfor further use or analysis or storage in a database 420 as describedabove.

It should be appreciated that such output data may have varied contentand be utilized in a variety of means according to the invention, andthat the arrangement illustrated is exemplary in nature and additionalor alternate components may be utilized according to the invention, suchas for further utilization of output data 460.

According to an embodiment of the invention, the analysis system has 5major steps:

-   -   1. INITALIZATION: of parking lot image data and raster        Car_Likelihood_Prior. The data input system consists of an image        header extraction function, an image display function, a        software cursor generator function controlled by means of a        trackball (or digital mouse or other interactive pointing        device) whose coordinates in the displayed image can be read        back into a processing function,    -   2. REGISTRATION: Given a pair of images for registration, a        tiepoint algorithm may be used on a given patch in a pair of        images in a temporal sequence. This will generate point        correspondences (with potentially many outliers) between the two        images        (x1,y1)i<->(x2,y2)i.

3. CHANGE DETECTION: Extraction of the Car_Temporal_Change functionseeks to answer, “Has there been a change between the two images?”

-   -   4. CAR DETECTION: Update raster values of Car_Likelihood_Prior        which signify the likelihood of a car/parking space existing at        each pixel [x y d]_(n) in the input digital image, and at a        particular orientation. Eight orientations may be used (from 0°        to 180°, each separated by 22.5°). The likelihoods may take        values on the unit interval [0.0, 1.0]    -   5. POST DETECTION OPTIMIZATION: Each car detection, provides a        location and orientation. A detector window may be applied over        every pixel in each patch. Many of these detections will overlap        one another, therefore a separate optimization, using a        Markov_Random_Field (MRF), on an arbitrary 2-D surface, removes        the overlaps. The surface has multiple nodes where each node is        a car detection with spatial neighbors. The “Data_Cost” is the        “cost of assigning a label to a particular node.” A        “Neighbor_Cost” is the “cost of assigning two labels over a        neighboring pair of nodes”. The MRF optimization is to find the        optimal labeling on the whole surface which minimizes the sum of        all “Data_Costs” and the sum of all “Neighbor_Costs”.

The following functions and/or algorithms require specificimplementation and parameter selection:

-   -   ParkingLot/RowDetection algorithm (Being developed under a        separate Patent Application)    -   raster Car_Likelihood_Prior function (Novel & unique data        storage and flags)    -   Car_Orientation function (Novel & unique s/w)    -   Car_Temporal_Change function (Novel & unique s/w)    -   Patch_Based_Epipolar_Geometry_Processing function (Novel &        unique s/w)    -   RANSAC (RANdom SAmple Consensus) algorithm for an iterative,        non-deterministic fitting function especially good at separating        inliers from outliers . . . (Prior Art s/w)    -   Sampson_Distance function (Prior Art s/w)    -   Semiglobal Optimization algorithm (Prior Art s/w)    -   Normalized Cross_Correlation function (Prior Art s/w)    -   Census_Transform_Cost function (Prior Art s/w)    -   Total_Variation_Optimization algorithm (Prior Art s/w)    -   Split_Bregman algorithm (Prior Art s/w)    -   Differential_Morphological_Profile (DMP) algorithm (Prior Art        s/w)    -   Tiepoint_Solution function (Prior Art s/w)    -   Top_Hat function (Prior Art s/w)    -   data output (Novel & unique s/w)    -   data storage/archiving function (Novel & unique s/w)    -   executive function to manage the overall process (Novel & unique        s/w)

According to a preferred embodiment of the invention, accurateregistration (down to the sub-pixel level) of temporal images isrequired for car detection and counting. Accurate multi-imageregistration of the same ground area is required for multiple purposessuch as:

Propagation/updating of the Raster Car_Likelihood_Prior function fromimage to image.

Extraction of the Car_Temporal_Change function.

Accounting for irregular temporal sampling intervals.

Ability to accommodate a large range of data acquisition parameters(i.e., satellite look angles and solar illumination geometry).

Ability to resolve the epipolar lines to sub-pixel accuracy.

In addition, this embodiment of the invention may make use of eitherpre-(Level 1B, 2A) or post-(Level 3A, 3D) Ortho-rectified image data:

Level 1B Images in sensor coordinates,

Level 2A Images projected onto an ellipsoid, i.e. height=0,

Level 3A Images projected onto low res Digital Elevation Map (DEM,100-200 m),

Level 3D Images projected onto higher res Digital Elevation Map (DEM,30-90 m).

An embodiment of the invention assumes that initial estimates areavailable for a small subset of car 2-D locations and orientations [xiyi di]_(n) along with viewing geometry, and solar illumination geometryfor a set of images of the same ground region or patch. Typically thesepatches are 1000-4000 pixels in extent, corresponding to approximately 2km by 2 km with resolution of 0.5 m-1.0 m.

According to an embodiment of the invention, a tiepoint algorithm may beused on a given patch in a pair of images in a temporal sequence. Thiswill generate point correspondences (with potentially many outliers)between the two images (x1, y1)i<->(x2,y2)i.

According to a preferred embodiment of the invention, when a new imagearrives in a temporal monitoring process, it is registered (usingregistration techniques such as those described herein) and comparedwith the previous image. According to an embodiment of the invention, itmay be shown that the image patches will not register over the entirepatch region because of a combination of errors in the remote sensinginstrument data acquisition parameters (e.g., sensor orientation,un-modeled atmospheric refraction), and errors in the Digital ElevationModel (DEM).

The former can be modeled with a Fundamental Matrix (F-matrix) thatapplies to the patch. The F-matrix is a 3×3 matrix, with # equal to thenumber of independent parameters, varying depending on type of remotesensing instrument used. For a calibrated frame camera #=5 parameters,for an uncalibrated frame camera #=7. For this case, we have #=4parameters. It may be shown by the embodiment of the invention that theremaining height error for any point lies on a line (hyperbola for apushbroom type instrument). For a tie-point (x1,y1)<->(x2,y2), given anF-matrix, where (x1,y1) in image #1, must have a match in image #2 whichlies on an epipolar line in image space given by the F-matrix. The samerelationship may be shown for (x2,y2).

Given a point (x1,y1) in image 1, the Sampson_Distance is the Euclideandistance in image space from a candidate match (x2,y2) in image 2 andthe epipolar line for (x1,y1) in image 1. To improve the optimizationfitting, a RANSAC process may be used to determine the F-matrix and theset of tie-points (inliers) for whom the Sampson_Distance may be <1pixel. Note that one need not solve for the 3D location of each andevery tie-point.

Another possible method would be to make use of a “bundle adjustment”where a nonlinear optimization may be implemented that simultaneouslysolves for the acquisition parameters and 3D location of all tiepoints.However, it may be shown that this nonlinear process requires extremelygood initialization and may be very sensitive to outliers: this is notthe case for the RANSAC process.

Lastly the Sampson_Distance may be calculated by:SD=Sqrt[{(a*x1)+(b*y1)+(c*x2)+(d*y2)+e}/sqrt{(a*a)+(b*b)+(c*c)+(d*d)}]

According to a preferred embodiment of the invention, once the F-matrixhas been determined, the sensor acquisition uncertainty may beestimated, and the images may be re-sampled so that the topographic(elevation) error is entirely represented by a shift in one dimension,typically along the remote sensing instrument x-axis.

An optimization is then done to compute the shift (disparity) at eachpixel using the SemiGlobal optimization process which implements eithera Normalized_Cross_Correlation or a Census_Transform_Cost function. ThisSemiGlobal optimization, instead of implementing a full 3-D optimizationover the set of all pixels in the patch, does at each pixel, anindependent optimization on azimuth lines in the imagery in eightdifferent directions (from 0° to 180° indexed by 22.5°), summing theresulting disparity likelihoods. The optimal disparity likelihood isthen selected at each pixel location.

Because the SemiGlobal optimization is not fully 3-D, it introducesartifacts that require an additional compensating optimization step,called a Total_Variation optimization with an L1 normalization (TV-L1).This optimization step favors piecewise constant disparity regions withabrupt discontinuities and may be solved using the Split_Bregmantechnique.

According to a preferred embodiment of the invention, when a new imagearrives in a temporal monitoring process, it is registered (usingregistration techniques such as those described herein in ¶¶[095]-[112])and compared with the previous image. A “change detection feature” isthen computed. Instead of using a pixel-based change detection method,which tends to not be robust, the embodiment implements a structuralmethod known in the art as Differential Morphological Profiles (DMP).DMP can be used to identify bright or dark areas of a particular size inan image.

Such an approach can be shown to be invariant to radiometric offsetsfrom the remote sensing system. DMP consists of applying opening andclosing spatial filter kernels with reconstruction to an image, and thensubtracting the result from the original image. This process is known inthe art as applying a Tophat function. According to a preferredembodiment of the invention, the process first identifies in both imagesall bright and dark objects that would be removed by a 6 m×6 m widekernel. Regions having contrast with their surround (as determined withthe Tophat operator) greater than some Threshold will be retained. Eachpixel is then labeled as (bright, dark, or background). At each pixel, achange is said to have occurred if the two images have differing labels,and the absolute radiometric difference between the original imagevalues is greater than the same DMP Threshold.

According to a preferred embodiment of the invention, an update is thenperformed to the raster Car_Likelihood_Prior, which increases thelikelihood uniformly over all directions at each pixel where a changehas occurred. In addition, a change mask is created for the new imagerepresenting three cases (1=bright object in image and change occurred,2=dark object in image and change occurred, 3=other). This change maskis then fed into the car detection step as a “feature image”.

According to an embodiment of the invention, in the art for a givensatellite or aerial image acquisition, there may be uncertainty in boththe instrument mounting location on the satellite and the instrumentangular orientation with respect to the satellite R,P,Y vectors. Themeasurement of these locations and angles is referred to in the art asMetrology. In the art, for either a framing camera or a push broomsensor, the metrological errors may be modeled by a 3-D positionaloffset and a 3-angle relative rotation. In the art for satellite remotesensing, if one assumes a near circular orbit, the positional errors asdescribed by a set of three linear offsets (which may be found by a dataextraction module 440), typically may be ignored. As a result, in theart, aligning a particular satellite image to an absolute coordinatesystem may be described with a 1×3 set of three rotational parameters.

In the art, with a satellite remote sensing instrument observing aground patch, imaging may be considered approximately Affine, which canbe mathematically expressed by a four parameter Fundamental Matrix. Inthe art, furthermore, an Affine imaging system, such as a remote sensingsystem, may be described by a linear approximation, such as a 1st orderTaylor series. The accuracy of the approximation may be found to beinversely proportional to the size of the ground patch over which it isapplied.

According to an embodiment of the invention, image tiepoints may beselected from two images of the same ground patch in a temporalsequence, and used in a RANSAC process to compute the optimalFundamental Matrix for the patch. Any common tiepoint approach may beused; e.g. normalized cross correlation, or SIFT (Scale-InvariantFeature Transform) descriptors, etc.

According to an embodiment of the invention, in the art, a 2-D imagetiepoint may be represented in homogeneous coordinates as [x y 1] T.Given this expression in the art, then the epipolar constraint may be:x1T*F*x2=0, where F may be the 3×3 Fundamental Matrix, and x1, x2, arethe corresponding 2-D image tiepoints in two separate, but overlappingimage patches.

Furthermore in the art, T is a vector transpose, so a 2-D point (x, y)in a patch is written as a 3-element vector [x y 1] T. In the art, theexpression Homogeneous means that the last parameter is a normalizationfactor such as (u,v)=(x/z, y/z)=[x y z] T=[x/z y/z 1] T. As embodied inthe invention, the expression in the art for a Fundamental Matrix maydescribe an Affine imaging system: F=[0 0 a; 0 0 b; c d e], which may befound to require five parameters, along with an arbitrary scale factor.According to an embodiment of the invention, with the F-matrix nowevaluated, the sensor acquisition uncertainty may become known. In anembodiment of the invention, this process may be extended withadditional tiepoint sets, within the patch, which will provide a betterestimation for the scale factor and decrease the sensor acquisitionuncertainty.

For an embodiment of the invention, as expressed in the art, given anF-matrix, then the image may be resampled and a rectifying Homographymay then computed for each image from the F-matrix, . . . ,x1Rectified=H1*x1, x2Rectified=H2*x2 for all pixels x1, y2 in theirrespective images. As expressed in the art, H1 and H2 are 3×3 matrices(homographies) also determined from the F-matrix. For an embodiment ofthe invention, this expression of the art may be found to distribute thematching uncertainty, due to the height error from the DEM, entirelyalong the x-axis.

For an embodiment of the invention, now solve for the 1-D shift(disparity) at each pixel. Semibglobal is a standard technique in theart for solving this type of optimization. Parameters may be selected sothat Semiglobal will compute 1-D optimizations over 8 azimuth directionsin the image (each separated by 22.5°). In this expression of the art,each direction may produce a likelihood of occurance for each disparityat a pixel. These likelihoods may be aggregated, or summed, and the mostlikely (i.e., optimal) one may be selected.

According to an embodiment of the invention, for each parking/storagefacility, estimates of the orientation, and 3-D location for eachpossible parking space or slot may need to be tracked over time. Theseparameters are assumed to be unchanging over time, but as moreinformation (i.e. contained in a time-ordered series of digital images)becomes available, these parameters may be re-estimated by methodsperformed by extraction module 440, thus improving the accuracy inparking space location. Finally, the number of cars in theparking/storage facility may be estimated from each digital image, andmonitored over a period of time by analysis of additional time-orderedseries of digital images.

Relative Rotation Estimation

According to an embodiment of the invention, estimating the threerelative rotation parameters can be accomplished using any number ofmodern computer vision and/or photogrammetry techniques. In the art, onesuch embodiment may be to determine image tiepoints (correspondences) ina time-ordered series of digital images, and then perform a “bundleadjustment”, to align the images with respect to one another. Such arelative alignment also yields improved alignment with a globalcoordinate system, as the pointing errors for multiple images may havelarge uncorrelated components across different orbits. In the art,tiepoint approaches such as those using interest points and descriptors(SURF, SIFT, etc.) may be used, as well as normalized_cross_correlationof image patches.

Initialization of Parking Lot and Raster Car_Likelihood_Prior

In the embodiment of the invention, raster Car_Likelihood_Prior,signifies the likelihood of a car/parking space existing at each pixelat a particular orientation. Eight orientations are used, separated by22.5°. The likelihoods take values on the unit interval [0.0, 1.0]. Theembodiment assumes polygonal areas of interest are given, e.g. a set ofparking lots. In addition, individual rows with car orientation mayoptionally be given. This data can be provided manually, throughexisting data sets such as OPENSTREETMAP™, or through automateddetection algorithms such as ParkingLot/RowDetection.

Registration

For the embodiment of this invention, accurate sub-pixel levelregistration of time-ordered images may be required for this applicationin several respects: propagation/updating of the Car_Likelihood_Priorfrom image-to-image, and extraction of temporal Change_Detectionfeatures. Registration requires two things: resolving the uncertainty inthe satellite acquisition parameters and correcting the local DEM(Digital Elevation Model) error.

An embodiment of the invention begins the Registration process byrunning a tiepoint algorithm which provides point correspondences (withpotentially many outliers) between two images:(x1i,y1i)<->(x2i,y2i)

The former may be accomplished using bundle adjustment, however for theembodiment of the invention an approach may be used which focuses solelyon resolving epipolar lines. The major advantages of doing this are torealize a substantial increase in computational speed (˜1000× fasterthan bundle adjustment), and to realize major improvements in theaccuracy of the epipolar geometry.

Prior art bundle adjustment software may not guarantee sub-pixelaccuracy in results, due to unresolved atmospheric parameters over thefull aperture of the remote sensing instrument, Chief among these beingatmospheric refractivity, and velocity aberration at optical and near IRwavelengths. Instead one of the key embodiments of this invention is towork over sub-regions of the remote sensing data frame referred toground patches (˜2 km×˜2 km) over which we it is possible to resolve theepipolar lines to sub-pixel accuracy.

Attempting to apply the embodiments of this invention over an entireimage will show that the images do not register because of a combinationof errors in data acquisition parameters, such as lack of accuracy inknowledge of the sensor position and orientation on the spacecraft, lackof accuracy in the pointing of the spacecraft, lack of accuracy inmodeled atmospheric refraction and potential height errors in theDigital Elevation Model. (DEM).

Errors in knowledge of metrology, may be modeled with aFundamental-Matrix applied separately to each ground patch. The Sampsondistance, in the art, can be computed directly from a FundamentalMatrix. For a satellite sensor observing a ground patch, the imaging canbe considered to be approximately Affine, resulting in a four parameterFundamental Matrix. In the art, given a point (x1,y1), the Sampsondistance is the Euclidean distance in image space from a candidatetiepoint match (x2,y2) and the epipolar line for (x1,y1). RANSAC in theart may be used to determine the F-matrix and the set of tiepoints(inliers) whose Sampson distance may be <1 pixel. Note that one need notsolve for the 3D location of all tiepoints, which is a major time saver.

Image tiepoints are used in the RANSAC process to compute the optimalFundamental Matrix for a patch. Any common tiepoint approach may be used(Normalized Cross Correlation, SIFT Descriptors, etc.). Representing a2-D image point in homogeneous coordinates as [x y 1]T, the epipolarconstraint is: x1T*F*x2=0, where F is the 3×3 Fundamental Matrix, andx1, x2, are the corresponding 2-D image points from two separate images.For Affine imaging,

-   -   F=[0 0 a; 0 0 b; c d e], which is five parameters, along with an        arbitrary scale factor.    -   T is a vector transpose, so a 2-D point (x, y) in an image is        written as a 3-vector [x y 1] T    -   Homogeneous means that the last parameter is a normalization        factor, i.e.,        (u,v)=(x/z,y/z)=[x y z]T=[x/z y/z 1]T

For F, using Matlab™ notation, which is in the art, and referring to theSampson distance described above, inspection may show that it isinvariant to the method by which parameters (a,b,c,d,e) may be scaled.Thus, there are only 4 independent parameters, selected to scale suchthat:a*a+b*b+c*c+d*d=1

In the art, the Epipolar constraint is therefore: x1*T*F*x2=0, fromwhich one can derive the Sampson distance.

In the art, one may compare a RANSAC optimization process to BundleAdjustment where you do a nonlinear optimization that simultaneouslysolves for the acquisition parameters and 3D locations of all tiepoints.This Bundle Adjustment process requires very good initialization and maybe sensitive to outliers, which may not be so for a RANSAC optimization.

Once the F-matrix may be estimated, the sensor acquisition uncertaintymay be known, and the remote sensing data can be re-sampled so that anytopographic error may be entirely represented by a shift in onedimension, typically the x-axis.

An optimization may then be done to compute the shift (disparity) ateach pixel. This can be done using any standard Photogrammetric stereoapproach. For the embodiment of this invention, Semiglobal optimizationwith either a Normalized_Cross_Correlation or a Census_Transform_Costfunction may be used. This optimization technique instead of doing afull 2-D optimization over the set of all pixels, does independent 1-Doptimizations on lines in the imagery along eight different azimuthdirections, summing the resulting disparity likelihoods. The optimalDisparity_Likelihood may then be selected at each pixel.

For an initial disparity d at each pixel (from Semiglobal), the finaldisparity f may be computed. A joint optimization over all pixels may bedone as:Sum[|di−fi|+lambda*|grad f|]

The Total_Variation is the L1_Norm of the spatial 2-D gradient of:f: |grad f|=sqrt((fi,j−fi−1,j)^2+(fi,j−fi,j−1)^2)

|d−f| says the output disparities should be similar to the input, butoutliers can occur (L1 Norm allows this, whereas L2 does not). TheTotal_Variation favors disparities that are locally (piecewise) constantor slowly varying, but again, the L1_Norm allows for abrupt spatialdiscontinuities (buildings, trees, etc.) Note that d,f can be vectorvalued at each pixel. For example one can fuse N images together atonce, where N>=2.

Once the F-matrix is known, the images are resampled: a rectifyingHomography is computed for each image from the F-matrix, ie

x1Rectified=H1*x1, x2Rectified=H2*x2 for all pixels x1, y2 in theirrespective images.

.H1 and H2 are 3×3 matrices (homographies) determined from the F-matrix.This puts the matching uncertainty from the height error entirely alongthe x-axis. The 1-D shift (disparity) at each pixel may be evaluated.

The Census_Transform_Cost function is also in the art. In thisembodiment of the invention, for a scalar remote sensing data set, and awindow of dimensions w, h around a pixel, form a vector of length w*h−1,for a w*h window, where each element has 3 possible values:{1, if x−x0>T;0, if |x−x0|<=T,−1, if x−x0<−T}

Where x0 is the center pixel value, x is the value of a pixel in thewindow but not at the center, and T is a threshold.

Because this optimization may not be fully 2-D, it has artifacts in it.These artifacts may be compensated for by a second optimization, whichinvolves a Total_Variation optimization from the art with an L1 Normalso in the art (TV-L1). This optimization favors piecewise constantdisparity regions with abrupt discontinuities. The optimization may besolved using the Split-Bregman technique.

Change Detection

TV_L1 and Split-Bregman, are standard techniques, well known in theprior art, but usually used in isolation. It is the inventor's beliefthat this may be the first time they have been used together, along withSemiglobal, for this type of image lattice analysis.

When a new image arrives in a temporal monitoring process, it may beregistered and compared with the previous image. A Change_Detectionfeature may then be computed. For the embodiment of this invention,instead of a pixel-based change approach, which tends to not be veryrobust, a structural method based on Differential_Morphological_Profiles(DMP) was implemented. DMP may identify bright or dark areas in an imageof a particular size. Such an approach may be invariant to RadiometricOffsets (but may be susceptible to Radiometric Gain changes).DMP_opening(N)=original image−size N morphological opening byReconstructionDMP_closing(N)=original image−size N morphological closing byReconstruction

In the art, for a structuring element of size N (circle diameter orsquare length), one applies an N×N filter to the image patch andsubtracts that output from the original image to detect structures ofsize N.

DMP_opening(N) gives bright areas, and DMP_closing(N) gives dark areas.Reconstruction may be a Connected_Component_Operator, so if a region hasany portion larger than the structuring element of size=N, then theentire region is kept—including all of its fine scale boundary detail.

DMP consists of applying opening and closing kernels with reconstructionto an image, and then subtracting the result from the original image(Top_Hat). For the embodiment of this invention, one first identifies inboth images all bright and dark objects that will be removed by a 6 mwide square filter kernel. Regions having contrast with their surround(determined with the Top_Hat operator) greater than a thresholdparameter will be retained.

The DMP threshold parameter may be learned by interactive processing andnot changed. In terms of normalization, images can either be calibratedto surface reflectance, or a histogram normalization may be applied tothe individual images, which gives them the same histogram distribution(standard image processing). Non-retained pixels are given a value (0)that the learning can key off of. The histogram normalization curve maybe: Brightest 1% of pixels mapped to top 10% of output range, the samefor the darkest 1%, and the middle 98% of pixels will be linearly mappedto the middle 80% of the output range.

Each pixel may then be labeled as (Bright, Dark, Background). At eachpixel, a change may then be said to have occurred, if the two imageshave differing labels, and the Absolute Radiometric Difference betweenthe original image data values may be greater than the same DMPthreshold. An update may then be performed to the rasterCar_Likelihood_Prior, which increases the likelihood uniformly over alldirections at each pixel where a change occurred. In addition, a changemask may be created for the new image representing three cases {brightobject in image with change occurring, dark object in image with changeoccurring, other}. This change mask may then be fed into the detectionstep as a feature image.

Car Detection

The Car Detection is done using a boosted cascade of features. AdaBoost,in the art, may be used. Each stage consists of a boosted set ofone-dimensional features. False alarms are reduced by a factor of 0.99at each stage, with four stages being used. Ground truth data is createdoff-line and used in supervised training. The image is rotated through aset of eight uniformly spaced angles (22.5°), and the detection appliedindependently to each rotated image. Rotating an image may be fast andeasily multithreaded. The images may be resampled with a reasonablebilinear kernel, which minimizes aliasing artifacts. The addition of asmall amount of spatial smoothing typically improves detection rates.

Features are standard machine-learning-terminology. They are the inputsto the classifier. The embodiment of this invention boosts up a set ofweak 1-D classifiers into a strong classifier. The embodiment may learnfrom 100,000 features. Only a single feature may be used by anindividual weak classifier. For the embodiment of this inventionfeatures are coded and loaded into a window surrounding each pixel. A15×7 window is used, although this is an implementation detail. Thewindow needs to be larger than 1 car, but not too large. The window longaxis may be parallel with the long dimension of the car. The window maybe in meters (7.5 m×3.5 m) converted to pixels depending on imageresolution. Interactive training with modest changes to window size willyield different classifiers, but nearly identical detection rates. Theset of features available are:

-   -   Haar features derived from the panchromatic image    -   summation windows on the temporal change image    -   summation windows on the DMP for the image (identifies small        bright and dark regions)    -   raster Car_Likelihood_Prior at the current orientation

An embodiment of the invention allows for Supervised Learning bycreating a training set of “positives” (cars with known locations anddirection) and “negatives” (locations with no cars). This proof test setallows the user to test out parameter sets for AdaBoost and otheralgorithm constants which may maximize the detection rate.

In addition, separate detectors may be trained for 8 different sunazimuth offsets with respect to the car orientation. The appropriatedetector is selected for the current orientation. Features (b-d) allowthe false alarms to be reduced dramatically from one detector stage tothe next (0.99 instead of a more typical 0.5).

Post Detection Optimization

Each detection, provides a location and orientation, and also anexpectation as to the approximate size of the vehicle: if length/widthratio ˜=2-2.5 m, then width ˜=2-2.5 m. Cars have a typical size range.Since the resolution of the remote sensing imagery is known (0.5 to 1.0meters/pixel), the inverse gives 2.0 to 1.0 pixels/meter. Rotations ofthe data do not change resolution.

The detector window is applied over every pixel in each patch. Allfeatures from the pixels in the window are considered by the detector.However, many of these detections will overlap one another spatially. Asa result, a separate optimization to estimate the most likely set ofphysically realizable detections will require a multi-dimensional binary(“to keep” or “not to keep” for each detection) of 2^N complexity, for Ndetections. The optimization is formulated as a Markov_Random_Field(MRF), which can be represented by Data_Costs at each detection as wellas Neighbor_Costs for each neighbor on an arbitrary 2-D surface.

This surface has nodes, where each node is a car detection. The surfacemay be embedded in a 2-D plane. Each node has spatial neighbors. Eachnode may be assigned a [0 1] label that indicates a valid detection, ornot. The “data cost” is the “cost of assigning a label to a particularnode” (ignoring all other nodes). A “neighbor cost” is the “cost ofassigning the two labels over a neighboring pair of nodes”. Theoptimization is to find the optimal labeling on the whole surface whichminimizes the sum of all “data costs” and “neighbor costs”.

For a given detection, all other detections within a certain radius areconsidered neighbors. This radius is large enough to include overlappingdetections as well as the immediately adjacent spaces in a row of parkedcars. The Data_Cost may be given by the Car_Likelihood_Prior fordetection at that pixel and orientation. For the Neighbor_Cost withoverlapping detections, the cost of detecting both may be set toinfinity (overlapping detections: 2 cars cannot occupy the same physicalspace, it is not physically realizable), whereas all other labelcombinations are set to a learned cost, c (all parameters may bedetermined by supervised learning). All nodes are detections.

What makes any one detection more likely than another may be theCar_Likelihood_Prior. For a detection, the direction and the typicalsize and/or shape of a car are known, so it may be possible to determineif they are overlapping. All parameters may be determined by supervisedlearning. For non-overlapping neighbors, the cost is also c, unless theneighbor is a candidate for being an adjacent car in the parking row, inwhich case the cost may be 0, if both are detected and the orientationsare identical, or c/2 if they differ by no more than +/−45°, and cotherwise. ‘Adjacent car candidate’ is defined as a neighbor whosedetection-center offset-vector from the current detection is =>45° fromthe detected orientation (+/−180°).

The MRF optimization is ‘non-submodular’, thereby preventing Max_Flowoptimizations from being used. MRF optimizations can be divided intosubmodular and ‘non-submodular’, with the former easily solved withgraph cut techniques, such as Max_Flow. Non-submodular is much harder todeal with. A positive detection is typically defined as any detectionwindow which intersects any part of the object, and a false alarm whenthere may be no intersection. Thus, one may have 3 detections/object andstill have 100% detection and no false alarms. For counting this isundesirable.

The MRF optimization step provides a unique, novel, and originalsolution not found elsewhere in the art. Parking Lots often have carspacked-in very tightly, so we get excessive numbers of overlappingdetections with different directions, etc. The MRF optimization resolvesthis in an ‘optimal’ way. Any appropriate non-sub-modular optimizationmay be utilized at this point, including Quadratic_Pseudo_BOolean(QPBO), Tree_Re_Weighting (TRW-S), Loopy_Belief_Propagation (LBP),Sequential_Greedy_Selection (SGS), etc.

Typically, around ⅔ of the detections are removed by this process.Finally, the optimized detections are used to update theCar_Likelihood_Prior. Updates are done using a cost_function withlearned parameters:P->alpha*P+(1−alpha)*lambda

Where P is the probability at a given pixel and orientation, alpha aparameter, and lambda a 3-valued indicator {0=no detection, 1=detection,½=detection at that pixel, but direction differs by +/−45°}.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

In the art, parameters may be extracted from the image data header foreach digital image by extraction module 440, providing, for example, atime of observation, an orbital position of the satellite observingplatform, and an orientation of the satellite observing platform(R=roll, P=pitch, & Y=yaw) with respect to an earth-centered coordinatesystem.

In the art, ancillary data from each digital image data header mayprovide image acquisition parameters so that image distance measurements(in pixels) may be scaled to absolute linear measurements. In the art,ancillary data from each digital image data header may also provide sunangle position data for the time corresponding to the center of thedigital image, from which analysis module 450 calculates shadowdirection, length, and shape for the digital images.

In the art, initial estimates are available for each car's location andorientation. These initial estimates may be manually determined fromother remote sensing sources such as aerial or satellite imagery, orextracted from the digital image through some other means, such as byuse of a track-ball controlled cursor 410.

According to an embodiment of the invention, an analysis module 450incrementally refines the initial estimates for each car to be located,using information extracted from a time-ordered series of remote sensingdigital images of parking lots. This set of image data may be assumed toinclude parking lots of interest, with irregular time sampling intervalsand a large range of image acquisition parameters.

According to an embodiment of the invention, FIG. 5 is a systemarchitecture diagram of an alternate image analysis system 500 with dataand analysis modules residing in one or more cloud systems. These cloudsystems may be addressable on a browser through the Internet 501 oranother communications network to data and analysis modules previouslyloaded into one or more cloud-based systems (i.e., remotely operated andaccessible via a network as previously described).

According to an embodiment, a cursor 411 may be used to make aninteractive selection of a parked car 502 and determine initial locationcoordinates and orientation for a car image 502. Then the cursor may beused to interactively determine initial values for the car length towidth ratio. According to an embodiment, extraction module 440 in thecloud extracts the image acquisition parameters and sun position angulardata from the image data header. An analysis module 450 uses thoseparameters together with the initial physical parameters for a car(length to width ratio, and 2-D location) determined by application ofan interactive cursor to determine the relative image rotation angle fora given car image.

According to an embodiment of the invention, an analysis module 450 inthe cloud incrementally refines the initial estimates for each car to belocated and whose orientation will be determined using informationextracted from a time-ordered series of digital images of the parkinglots loaded into the cloud. A time-ordered series of imagery may beassumed to include parking lots of interest, with irregular timesampling intervals and a large range of image acquisition parameters. Inthe art, parameters may be extracted from the image data header for eachdigital image by data extraction module 440, providing, for example, atime of observation, an orbital position of the satellite observingplatform, and an orientation of the satellite observing platform(R=roll, P=pitch, & Y=yaw) with respect to an earth-centered coordinatesystem.

In the art, the ancillary data for each digital image may also provideimage acquisition parameters in order that image distance measurements(in pixels) may be scaled to absolute linear measurements. Ancillarydata for each digital image may also provide sun angle data for the timecorresponding to the center of the digital image, from which analysismodule 450 calculates shadow direction, and shape for the digital imagesof the parking lots.

In the art, initial estimates may be available for parking lot location,parking space orientation, and parking space width. These initialestimates may be provided manually, through existing data sets such asOPENSTREETMAP™, or through automated detection algorithms such asParkingLot/RowDetection. In the art, Ground truth data is createdoff-line and used in supervised learning or training. In the art onecreates a training set of positives (cars with knownlocations/directions) and negatives (no cars). Then by using these testimages, one may learn all the parameters (for Boosting and otheralgorithm parameters) that maximize both the change_detection rate andthe car_detection rate.

According to an embodiment of the invention, an analysis module 450incrementally refines the initial estimates for each parking lot spaceto be measured, using information extracted by extraction module 440from a time-ordered series of digital images of the parking lot. Atime-ordered series of imagery may be assumed to include parking lotspaces of interest, with irregular time sampling intervals and a largerange of image acquisition parameters.

According to an embodiment of the invention, system 600 (or system 500,as the basic process steps used in system 600 are analogous to those insystem 500) may be nested in a semi-automatic process. FIG. 6 is amethod diagram illustrating an exemplary method 600 for locating andanalyzing parking spaces within an image, according to an embodiment ofthe invention. The method assumes that a sufficient number of tiepointsin a sufficient number of patches in a pair of images have been locatedand the Registration process carried out prior to step 601.

Next, in step 602, individual parking spaces in a parking lot in thefirst image are located. Then the process determines for each parkingspace all parameters needed to uniquely identify the space and itsneighbor spaces.

In step 603 the process uses Car_Detection to determine if a car may beparked in the space, and in step 604 if the adjacent spaces empty orfull.

In a next step 605, the process determines the 2-D location,width-to-length ratio, orientation and updates Car_Likelihood_Prior.

In a next step 606, a car's characteristics may be used to determine ifthere is overlap with an adjacent parking space. If there is overlap, itmay be flagged in CarLikelihood_Prior.

The process continues with step 607 moving along rows locatingadditional parking spaces, and if they have cars in them or not.Additional images in the sequence are loaded and analyzed until theprocess is concluded by user input, or a processing flag is generated,halting the process for user input.

If no cars remain in the image area, in a final step 607 processed datamay be produced as output to be viewed or further processed or stored.

In the art, initial estimates may be available for parking lot location,parking space orientation, and parking space width. These initialestimates may be provided manually, through existing data sets such asOPENSTREETMAP™, or through automated detection algorithms such asparking lot/row detection. In an embodiment of the invention, rasterCar_Likelihood_Prior, signifies the likelihood of a car/parking spaceexisting at each pixel at a particular orientation. Eight orientationsare used, separated by 22.5°. The likelihoods take values on the unitinterval [0.0, 1.0]. The embodiment assumes polygonal areas of interestare given, e.g. a set of parking spaces. In addition, individual rowswith car orientation may optionally be given.

According to an embodiment of the invention, FIG. 7 is a method diagramillustrating an exemplary method 700 for locating and analyzing parkingspaces within an image and then determining if the spaces are occupiedby cars.

In an initial step 701, image data in a time ordered sequence may bereceived for analysis. An embodiment of the invention begins theRegistration process by running a tiepoint algorithm which providespoint correspondences (with potentially many outliers) between twoimages:(x1i,y1i)<->(x2i,y2i)

For a satellite sensor observing a ground patch, the remote sensingsystem can be considered to be approximately Affine, resulting in a fourparameter Fundamental_Matrix. In the art, given a point (x1,y1), theSampson distance is the Euclidean distance in image space from acandidate tiepoint match (x2,y2) to the epipolar line for (x1,y1). Instep 702 the RANSAC process, in the art, may be used to determine theF-matrix and the set of tiepoints (inliers) for whom theSampson_distance may be <1 pixel.

Image tiepoints are used in the RANSAC process to compute the optimalFundamental_Matrix, (in the art), for a patch. Any common tiepointapproach may be used (Normalized Cross Correlation, SIFT Descriptors,etc.). Representing a 2-D image point in homogeneous coordinates as [x y1]T, the epipolar constraint is: x1T*F*x2=0, where F is the 3×3Fundamental Matrix, and x1, x2, are the corresponding 2-D image pointsfrom two separate images. For Affine imaging,

F=[0 0 a; 0 0 b; c d e], which is five parameters, along with anarbitrary scale factor.

T is a vector transpose, so a 2-D point (x, y) in an image is written asa 3-vector [x y 1] T

Homogeneous means that the last parameter is a normalization factor,ie.,(u,v)=(x/z,y/z)=[x y z]T=[x/z y/z 1]T

In step 703, for F, using Matlab™ notation, which is in the art, andreferring to the Sampson_distance described above, which is also in theart, inspection shows that it is invariant to the method by whichparameters (a,b,c,d,e) may be scaled. Thus, there are only 4 independentparameters, selected so as to scale such that:a*a+b*b+c*c+d*d=1

In the art, the Epipolar constraint is therefore: x1*T*F*x2=0, fromwhich one can derive the Sampson_distance for the patch as:SD=sqrt[{(a*x1)+(b*y1)+(c*x2)+(d*y2)+e}/sqrt{(a*a)+(b*b)+(c*c)+(d*d)}]

In step 704, according to a preferred embodiment of the invention, oncethe Sampson_distance and the F_matrix have been determined, the sensoracquisition uncertainty may be estimated, and the images may bere-sampled so that any topographic (elevation) error is entirelyrepresented by a shift in one dimension, typically along the remotesensing instrument x-axis.

In step 705, an optimization is first done to compute the shift(disparity) at each pixel using the SemiGlobal optimization processwhich makes use of either a Normalized_Cross_Correlation or aCensus_Transform_Cost function. This SemiGlobal optimization, instead ofimplementing a full 3-D optimization over the set of all pixels in thepatch, does at each pixel, an independent optimization on azimuth linesin the imagery in eight different directions (from 0° to 180°, indexedby 22.5°), summing the resulting disparity_likelihoods. The optimaldisparity_likelihood is then selected at each pixel location.

Because the SemiGlobal optimization is not fully 3-D, it introducesartifacts that require an additional compensating optimization step 706,called a Total_Variation optimization with an L1 normalization (TV-L1).This optimization step favors piecewise constant disparity regions withabrupt discontinuities and may be solved using the Split_Bregmantechnique

According to a preferred embodiment of the invention, when a new imagearrives in a temporal monitoring process, it is registered (usingregistration techniques such as those disclosed herein) and comparedwith the previous image. A “change detection feature” is then computed.Instead of using a pixel-based change detection method, which tends toproduce a large amount of noise, the embodiment implements, in step 707,a lattice structural method known in the art as DifferentialMorphological Profiles (DMP). DMP can be used to identify bright or darkareas of a particular size in an image.

DMP consists of applying opening and closing spatial filter kernels withreconstruction to an image, and then subtracting the result from theoriginal image. This process is known in the art as applying a Tophatfunction. According to a preferred embodiment of the invention, theprocess first identifies in both images all bright and dark objects thatwould be removed by a 6 m×6 m wide kernel. Regions having contrast withtheir surround (as determined with the Tophat operator) greater thansome Threshold will be retained. In step 708, each pixel is then labeledas (bright, dark, or background). At each pixel, a change is said tohave occurred if the two images have differing labels, and the absoluteradiometric difference between the original image values is greater thanthe same DMP Threshold.

According to a preferred embodiment of the invention, in step 709 anupdate is then performed to the raster Car_Likelihood_Prior, whichincreases the likelihood uniformly over all directions at each pixelwhere a change has occurred. In addition, a change mask is created forthe new image representing three cases (1=bright object in image andchange occurred, 2=dark object in image and change occurred, 3=other).This change mask is then fed into the Car Detection step as a “featureimage”.

In the embodiment of the invention, the Car Detection may be done instep 710 using a boosted cascade of features. AdaBoost, in the art, maybe used. Each stage consists of a boosted set of one-dimensionalfeatures. False alarms are reduced by a factor of 0.99 at each stage,with four stages being used. The image is rotated through a set of eightuniformly spaced angles (22.5°) from 0° to 180°, and the detectionapplied independently to each rotated image. The images may be resampledwith a reasonable bilinear kernel, which minimizes aliasing artifacts.The addition of a small amount of spatial smoothing typically improvesdetection rates.

In step 711, for the embodiment of this invention, features are codedand loaded into a window surrounding each pixel. A 15×7 window is used.The window needs to be larger than 1 car, but not too large. The windowlong axis should parallel the long dimension of the car. The set offeatures available are:

-   -   Haar features    -   summation windows on the temporal change image    -   summation windows on the DMP for the image (identifies small        bright and dark regions)    -   raster Car_Likelihood_Prior at the current orientation

According to an embodiment of the invention, FIG. 8 is a method diagramillustrating an exemplary method 800 for Post Detection Optimization ofcars in parking spaces within a remote sensed image. The method assumesthat in step 801 an image has been down loaded and undergone tie-pointregistration.

In step 802, in an embodiment of the invention, the detector window isapplied over every pixel in each patch. All features from the pixels inthe window are considered by the detector. However, many of thesedetections will overlap one another spatially. As a result, a separateoptimization to estimate the most likely set of physically realizabledetections will require a multi-dimensional binary (“to keep” or “not tokeep” for each detection) of 2^N complexity, for N detections. Theoptimization is formulated as a Markov_Random_Field (MRF), which can berepresented by Data_Costs at each detection as well as Neighbor_Costsfor each neighbor on an arbitrary 2-D surface.

In a next step 803, in the embodiment of the invention, this surface hasnodes, where each node is a car detection. Each node has spatialneighbors. Each node may be assigned a [0 1] label that indicates avalid detection, or not. The “data cost” is the “cost of assigning alabel to a particular node” (ignoring all other nodes). A “neighborcost” is the “cost of assigning the two labels over a neighboring pairof nodes”. The MRF optimization is to find the optimal labeling on thewhole surface which minimizes the sum of all “data costs” and the sum ofall “neighbor costs”.

In step 804, in the embodiment of the invention, for a given detection,all other detections within a certain radius are considered neighbors.This radius may be large enough to include overlapping detections aswell as the immediately adjacent spaces in a row of parked cars. TheData_Cost may be given by the Car_Likelihood_Prior for detection at thatpixel and orientation. For the Neighbor_Cost with overlappingdetections, the cost of detecting both may be set to infinity (foroverlapping detections: 2 cars cannot occupy the same physical space, itis not physically realizable), whereas all other label combinations areset to a learned cost, c (all parameters may be determined by supervisedlearning). All nodes are detections.

In step 805, in the embodiment of the invention, what makes any onedetection more likely than another may be the Car_Likelihood_Prior. Fora detection, the direction and the typical size and/or shape of a carare known, so it may be possible to determine if they are overlapping.All parameters may be determined by supervised learning. Fornon-overlapping neighbors, the cost is also c, unless the neighbor is acandidate for being an adjacent car in the parking row, in which casethe cost may be 0, if both are detected and the orientations areidentical, or c/2 if they differ by no more than +/−45°, and cotherwise. ‘Adjacent car candidate’ is defined as a neighbor whosedetection-center offset-vector from the current detection is =>45° fromthe detected orientation (+/−180°).

In a next step 806, in the embodiment of the invention, the MRFoptimization is ‘non-sub-modular’, thereby preventing Max_Flowoptimizations from being used. MRF optimizations can be divided intosub-modular and ‘non-sub-modular’, with the former easily solved withgraph cut techniques, such as Max_Flow. A non-sub-modular condition maybe much harder to deal with. A positive detection is typically definedas any detection window which intersects any part of the object, and afalse alarm when there may be no intersection. Thus, one may have 3detections/object and still have 100% detection and no false alarms. Forcounting this is undesirable.

In a next step 807, in the embodiment of the invention, the MRFoptimization process provides a unique, novel, and original solution notfound elsewhere in the art. Parking Lots often have cars packed-in verytightly, so there may be excessive numbers of overlapping detectionswith different directions, etc. The MRF optimization resolves this in an‘optimal’ way. Any appropriate non-sub-modular optimization may beutilized at this point, including Quadratic_Pseudo_BOolean (QPBO),Tree_Re_Weighting (TRW-S), Loopy_Belief Propagation (LBP),Sequential_Greedy_Selection (SGS), etc.

In the next step 808, in the embodiment of the invention, around ⅔ ofthe detections may be removed by the MRF optimization process. Finally,the optimized detections are used to update the Car_Likelihood_Prior.Updates are done using a cost_function with learned parameters:P->alpha*P+(1−alpha)*lambda

Where P is the probability at a given pixel and orientation, alpha aparameter, and lambda a 3-valued indicator {0=no detection, 1=detection,½=detection at that pixel, but direction differs by +/−45°}.

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

What is claimed is:
 1. A system for automated remote car counting, the system comprising: a satellite-based image collection subsystem; a data storage subsystem comprising a database of images obtained from the satellite-based image collection subsystem; and an analysis software module stored and operating on computer coupled to the data storage subsystem; wherein the satellite-based image collection subsystem collects a plurality of images corresponding to a plurality areas of interest and stores them in the data storage subsystem; and wherein the analysis module carries out the following steps to compute the number of cars in an area of interest: (a) retrieving an image corresponding to an area of interest from the data storage subsystem; (b) identifying a parking space in the image; (c) determining if there is a car located in the parking space; (d) determining location, size, and angular direction of a car in a parking space; (e) determining an amount of overlap of a car with an adjacent parking space; (f) iterating steps (b)-(e) until no unprocessed parking spaces remain; and (g) iterating steps (a)-(f) until no unprocessed images corresponding to areas of interest remain.
 2. A method of automatically counting cars, the method comprising the steps of: (a) retrieving, using a satellite-based image collection subsystem, a plurality of images and storing them in a data storage subsystem; (b) retrieving, using an analysis software module stored and operating on computer coupled to the data storage subsystem, an image corresponding to an area of interest from the data storage subsystem; (c) identifying, using the analysis software module, a parking space in the image; (d) determining if there is a car located in the parking space; (e) determining a location, size, and angular direction of a car in a parking space; (f) determining an amount of overlap of a car with an adjacent parking space; (g) iterating steps (b)-(e) until no unprocessed parking spaces remain; and (h) iterating steps (a)-(f) until no unprocessed images corresponding to areas of interest remain. 